COVID-19 Prognostic Models: A Pro-con Debate for Machine Learning vs. Traditional Statistics.

The SARS-CoV-2 virus, which causes the COVID-19 pandemic, has had an unprecedented impact on healthcare requiring multidisciplinary innovation and novel thinking to minimize impact and improve outcomes. Wide-ranging disciplines have collaborated including diverse clinicians (radiology, microbiology, and critical care), who are working increasingly closely with data-science. This has been leveraged through the democratization of data-science with the increasing availability of easy to access open datasets, tutorials, programming languages, and hardware which makes it significantly easier to create mathematical models. To address the COVID-19 pandemic, such data-science has enabled modeling of the impact of the virus on the population and individuals for diagnostic, prognostic, and epidemiological ends. This has led to two large systematic reviews on this topic that have highlighted the two different ways in which this feat has been attempted: one using classical statistics and the other using more novel machine learning techniques. In this review, we debate the relative strengths and weaknesses of each method toward the specific task of predicting COVID-19 outcomes.

COVID-19 Prognostic Models: A Pro-con Debate for Machine Learning vs. Traditional Statistics

The SARS-CoV-2 virus, which causes the COVID-19 pandemic, has had an unprecedented impact on healthcare requiring multidisciplinary innovation and novel thinking to minimize impact and improve outcomes. Wide-ranging disciplines have collaborated including diverse clinicians (radiology, microbiology, and critical care), who are working increasingly closely with data-science. This has been leveraged through the democratization of data-science with the increasing availability of easy to access open datasets, tutorials, programming languages, and hardware which makes it significantly easier to create mathematical models. To address the COVID-19 pandemic, such data-science has enabled modeling of the impact of the virus on the population and individuals for diagnostic, prognostic, and epidemiological ends. This has led to two large systematic reviews on this topic that have highlighted the two different ways in which this feat has been attempted: one using classical statistics and the other using more novel machine learning techniques. In this review, we debate the relative strengths and weaknesses of each method toward the specific task of predicting COVID-19 outcomes.

Clinical Utility and Functionality of an Artificial Intelligence-Based App to Predict Mortality in COVID-19: Mixed Methods Analysis.

BACKGROUND: The artificial neural network (ANN) is an increasingly important tool in the context of solving complex medical classification problems. However, one of the principal challenges in leveraging artificial intelligence technology in the health care setting has been the relative inability to translate models into clinician workflow. OBJECTIVE: Here we demonstrate the development of a COVID-19 outcome prediction app that utilizes an ANN and assesses its usability in the clinical setting. METHODS: Usability assessment was conducted using the app, followed by a semistructured end-user interview. Usability was specified by effectiveness, efficiency, and satisfaction measures. These data were reported with descriptive statistics. The end-user interview data were analyzed using the thematic framework method, which allowed for the development of themes from the interview narratives. In total, 31 National Health Service physicians at a West London teaching hospital, including foundation physicians, senior house officers, registrars, and consultants, were included in this study. RESULTS: All participants were able to complete the assessment, with a mean time to complete separate patient vignettes of 59.35 (SD 10.35) seconds. The mean system usability scale score was 91.94 (SD 8.54), which corresponds to a qualitative rating of "excellent." The clinicians found the app intuitive and easy to use, with the majority describing its predictions as a useful adjunct to their clinical practice. The main concern was related to the use of the app in isolation rather than in conjunction with other clinical parameters. However, most clinicians speculated that the app could positively reinforce or validate their clinical decision-making. CONCLUSIONS: Translating artificial intelligence technologies into the clinical setting remains an important but challenging task. We demonstrate the effectiveness, efficiency, and system usability of a web-based app designed to predict the outcomes of patients with COVID-19 from an ANN.

Bacteraemia variation during the COVID-19 pandemic; a multi-centre UK secondary care ecological analysis.

BACKGROUND: We investigated for change in blood stream infections (BSI) with Enterobacterales, coagulase negative staphylococci (CoNS), Streptococcus pneumoniae, and Staphylococcus aureus during the first UK wave of SARS-CoV-2 across five London hospitals. METHODS: A retrospective multicentre ecological analysis was undertaken evaluating all blood cultures taken from adults from 01 April 2017 to 30 April 2020 across five acute hospitals in London. Linear trend analysis and ARIMA models allowing for seasonality were used to look for significant variation. RESULTS: One hundred nineteen thousand five hundred eighty-four blood cultures were included. At the height of the UK SARS-CoV-2 first wave in April 2020, Enterobacterales bacteraemias were at an historic low across two London trusts (63/3814, 1.65%), whilst all CoNS BSI were at an historic high (173/3814, 4.25%). This differed significantly for both Enterobacterales (p = 0.013), CoNS central line associated BSIs (CLABSI) (p < 0.01) and CoNS non-CLABSI (p < 0.01), when compared with prior periods, even allowing for seasonal variation. S. pneumoniae (p = 0.631) and S. aureus (p = 0.617) BSI did not vary significant throughout the study period. CONCLUSIONS: Significantly fewer than expected Enterobacterales BSI occurred during the UK peak of the COVID-19 pandemic; identifying potential causes, including potential unintended consequences of national self-isolation public health messaging, is essential. High rates of CoNS BSI, with evidence of increased CLABSI, but also likely contamination associated with increased use of personal protective equipment, may result in inappropriate antimicrobial use and indicates a clear area for intervention during further waves.

Comparison of deep learning with regression analysis in creating predictive models for SARS-CoV-2 outcomes.

BACKGROUND: Accurately predicting patient outcomes in Severe acute respiratory syndrome coronavirus 2 (SARS-CoV-2) could aid patient management and allocation of healthcare resources. There are a variety of methods which can be used to develop prognostic models, ranging from logistic regression and survival analysis to more complex machine learning algorithms and deep learning. Despite several models having been created for SARS-CoV-2, most of these have been found to be highly susceptible to bias. We aimed to develop and compare two separate predictive models for death during admission with SARS-CoV-2. METHOD: Between March 1 and April 24, 2020, 398 patients were identified with laboratory confirmed SARS-CoV-2 in a London teaching hospital. Data from electronic health records were extracted and used to create two predictive models using: (1) a Cox regression model and (2) an artificial neural network (ANN). Model performance profiles were assessed by validation, discrimination, and calibration. RESULTS: Both the Cox regression and ANN models achieved high accuracy (83.8%, 95% confidence interval (CI) 73.8-91.1 and 90.0%, 95% CI 81.2-95.6, respectively). The area under the receiver operator curve (AUROC) for the ANN (92.6%, 95% CI 91.1-94.1) was significantly greater than that of the Cox regression model (86.9%, 95% CI 85.7-88.2), p = 0.0136. Both models achieved acceptable calibration with Brier scores of 0.13 and 0.11 for the Cox model and ANN, respectively. CONCLUSION: We demonstrate an ANN which is non-inferior to a Cox regression model but with potential for further development such that it can learn as new data becomes available. Deep learning techniques are particularly suited to complex datasets with non-linear solutions, which make them appropriate for use in conditions with a paucity of prior knowledge. Accurate prognostic models for SARS-CoV-2 can provide benefits at the patient, departmental and organisational level.

Investigating the association between ethnicity and health outcomes in SARS-CoV-2 in a London secondary care population.

BACKGROUND: Black, Asian and minority ethnic (BAME) populations are emerging as a vulnerable group in the severe acute respiratory syndrome coronavirus disease (SARS-CoV-2) pandemic. We investigated the relationship between ethnicity and health outcomes in SARS-CoV-2. METHODS AND FINDINGS: We conducted a retrospective, observational analysis of SARS-CoV-2 patients across two London teaching hospitals during March 1 -April 30, 2020. Routinely collected clinical data were extracted and analysed for 645 patients who met the study inclusion criteria. Within this hospitalised cohort, the BAME population were younger relative to the white population (61.70 years, 95% CI 59.70-63.73 versus 69.3 years, 95% CI 67.17-71.43, p<0.001). When adjusted for age, sex and comorbidity, ethnicity was not a predictor for ICU admission. The mean age at death was lower in the BAME population compared to the white population (71.44 years, 95% CI 69.90-72.90 versus, 77.40 years, 95% CI 76.1-78.70 respectively, p<0.001). When adjusted for age, sex and comorbidities, Asian patients had higher odds of death (OR 1.99: 95% CI 1.22-3.25, p<0.006). CONCLUSIONS: BAME patients were more likely to be admitted younger, and to die at a younger age with SARS-CoV-2. Within the BAME cohort, Asian patients were more likely to die but despite this, there was no difference in rates of admission to ICU. The reasons for these disparities are not fully understood and need to be addressed. Investigating ethnicity as a clinical risk factor remains a high public health priority. Studies that consider ethnicity as part of the wider socio-cultural determinant of health are urgently needed.

Prognostic Modeling of COVID-19 Using Artificial Intelligence in the United Kingdom: Model Development and Validation.

BACKGROUND: The current severe acute respiratory syndrome coronavirus 2 (SARS-CoV-2) outbreak is a public health emergency and the case fatality rate in the United Kingdom is significant. Although there appear to be several early predictors of outcome, there are no currently validated prognostic models or scoring systems applicable specifically to patients with confirmed SARS-CoV-2. OBJECTIVE: We aim to create a point-of-admission mortality risk scoring system using an artificial neural network (ANN). METHODS: We present an ANN that can provide a patient-specific, point-of-admission mortality risk prediction to inform clinical management decisions at the earliest opportunity. The ANN analyzes a set of patient features including demographics, comorbidities, smoking history, and presenting symptoms and predicts patient-specific mortality risk during the current hospital admission. The model was trained and validated on data extracted from 398 patients admitted to hospital with a positive real-time reverse transcription polymerase chain reaction (RT-PCR) test for SARS-CoV-2. RESULTS: Patient-specific mortality was predicted with 86.25% accuracy, with a sensitivity of 87.50% (95% CI 61.65%-98.45%) and specificity of 85.94% (95% CI 74.98%-93.36%). The positive predictive value was 60.87% (95% CI 45.23%-74.56%), and the negative predictive value was 96.49% (95% CI 88.23%-99.02%). The area under the receiver operating characteristic curve was 90.12%. CONCLUSIONS: This analysis demonstrates an adaptive ANN trained on data at a single site, which demonstrates the early utility of deep learning approaches in a rapidly evolving pandemic with no established or validated prognostic scoring systems.